Empirical Bayes Screening for Link Analysis

The domain of link analysis has recently re-ignited interest among researchers due to its applicability to new areas such as intelligence analysis (for example, identifying cliques of suspicious people), large scale social network analysis and genomics. The area of link analysis is not new and comprise a number of techniques developed by different communities. In this paper we propose a statistical approach to answering questions such as: what would be the “interesting” k-tuples of entities (that can be people, ingredients in a recipe, etc depending on the application), given a dataset of observed ntuples of entities. A typical example of an n-tuple might be a set of people observed to be having a meeting, or observed traveling to the same destination. Currently, it is common to work with pairwise count matrices. Empirical Bayes Screening (EBS) has several advantages over existing methods, one of them being the ability to take advantage of the interactions of higher order (for example, a group of three people significantly working together even though no two of them have significantly atypical pairwise interaction). EBS has the additional advantage of being insensitive to the small sample size of co-occurrences. We discuss advantages and disadvantages of the algorithm and provide performance analysis based on several datasets.